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¢ Employee-Owned Small Business, aah 
Cambridge, MA, USA started in 1983 
e 150+ employees and associates 
e GSA Schedule for IT Services 


Bridging the gap between thought leaders 
in academia and systems developers/ 


Integ rators Software Developers 
¢ Long-term collaborative partner with world-class Years of Experience 
universities 
e Strong participation in professional societies/ 
panels Over 
. . ; 10 Years 
e Close relationships with several lead system 42% 
integrators 
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Core Company Competencies 


Sensor Processing & Networking 
e Automatic target recognition (ATR) and tracking 

e Automated analysis of security and surveillance imagery 
e Network and software trustworthiness optimization 

e Vision-based autonomous navigation 

e Image and video enhancement 


Decision Management 
¢ Data mining, data/information fusion 
e Event detection and prediction 
e Situation analysis and threat assessment 
e Sensor/resource management 
e Course-of-action analysis 
e Cyber security 


Socio-Cognitive Systems 
¢ Sensor-based assessment of human physical and cognitive states 
¢ Human behavior modeling for individuals, organizations, 

and societies 
¢ Model-driven and model-backed assessment of complex situations 
| ¢ Tactical intelligence management and collection 


Human Effectiveness 
e Intuitive interfaces and operator support tools for complex systems 
e Work analysis, design prototyping, and human-in-the-loop evaluation 
¢ Skill modeling, intelligent tutoring, game- and simulation- 
based training 
¢ Information visualization 
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Charles River bridges the gap between universities’ basic research 
and lead system integrators’ needs for mature technology 


Universities 

Arizona State University 

Brandeis University 

Brown University 

Carnegie Mellon University 

Cornell University 

Claremont Graduate 
University 

Duke University 

George Mason University 

Harvard University 

Harvard Medical School 

Massachusetts Inst. of 
Technology 

Penn State University 

Syracuse University 

Tufts University 

University at Buffalo (SUNY) 

University of California at 
Berkeley 

University of Illinois 

University of Kentucky 

University of Maryland 

University of Massachusetts 

University of Massachusetts 
Medical School 

University of Pennsylvania 

University of Southern California 

Vanderbilt Medical School 

Wright State University 


Other Businesses 


Anacapa Sciences, Inc. 

Analytic Graphics, Inc. 
(AGI) 

Apelon, Inc. 

Aptima, Inc. 

Architecture Technology 
Corp (ATC) 

Assured Information 
Security, Inc. 


Aviation Specialties, Inc. 


Biosignals PluxX 

Blue Force LLC 

Blyth Consulting Group 

Boston VA Research 
Institute 

Boundariez Consulting 

Clark & Parsia 

Coolhead Consulting 

Creation Logic 

CUBRC 

Data Fusion & Neural 
Networks 

Defense Health Program 

Global Training, LLC 

Groupi Solutions 

InGuardians 


Karabaich Strategic Info 
Services 

Massachusetts Eye and 
Ear Infirmary 

Massachusetts General 
Hospital 

McLean Hospital 

PAR Government 
Systems 

PatchPlus Consulting 

Planet 9 Studios 

Referentia Systems 
Incorporated 

Roth Cognitive 
Engineering 

Sensimetrics Corporation 

Signet Research & 
Consulting 

Singular Computing LLC 

SofCheck 

Spaulding Research 
Hospital 

Tech Projects, LLC 

Technology Solution 
Experts 

TopCoder 

Zanol Consulting, Inc. 


Large System Integrators 
Applied Research Associates 


Aurora Flight Sciences 

BAE Systems 

Boeing 

Booz Allen Hamilton 

Charles Stark Draper Laboratory 

Chenega Federal Systems 

DSS Lab 

General Dynamics 

Lockheed Martin 

L-3 Communications 

Metron Scientific Solutions 

MIT Lincoln Labs 

MITRE 

Northrop Grumman Corp 

Oculus Innovative Sciences 

Pacific Northwest National 
Laboratory 

QinetiQ North America 
(Foster-Miller) 

Raytheon 

Science Applications 
International Corp. (SAIC) 

Teledyne-Brown Engineering 

Textron Defense Systems 
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Powerful Productivity Tools 


Sherlock™ 


VisionKit® 


BNet® 


Connect™ 


AgentWorks™ 


Metronome™ 


DRIVE™ 


Figaro™ 


Open, extensible software and hardware platform for rapid prototyping 
of solutions to collect, analyze, visualize, and reason about human 
physiological, neurological, and behavioral state 


A library of computer vision software components for developing object 
recognition, surveillance, video enhancement, and visual navigation systems 


Bayesian Belief Network development tool and accompanying software 
development kit 


Social and Organization Network Analysis tool for domain-independent network 
analysis 


Intelligent agent toolkit providing agent-based computational components and 
integrated system development, validation, and visualization environments 


A plug-in based architecture and associated visual software components enabling 
developers of rich-client applications to create powerful, easy-to-use interfaces 
for high-availability and mission critical systems 


Toolkit to support rapid generation, reconfiguration, evaluation, and refinement of 
Information visualization content and format 


A language and reasoning framework for probabilistic modeling and inference 
that provides the basis for many computational intelligence services such as data 
and information fusion, machine learning, forecasting, state estimation, and 


sensitivity analysis 
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Human State Assessment Portfolio 


Completed 


Area 


Cognitive Workload Assessment 


Condition Detection 


Project 


ADAPTER II 


MEDIC II 


CAPT PICARD 
II 


ASYMPTOTE 


SEAHAWK 


SOMBA 


ADVISOR II 


READ-IT 


SERENE 


Customer 


USAF/AFRL 


CDMRP 


NASA 


USAF/AFRL 


US Navy 


DARPA 


AARL/MRMC 


ARI 


ONR 


Title 


Handheld Intelligent Tutors 


Personalized for Individual Training 


Aiding Training by Monitoring, 
Extracting, & Decoding Indicators 
of Cognitive Load 


Precision Information Environment 


for Coordinated Emergency 
Support 


Adaptable System for Measuring 
Performance of Teams of 
Operators in Targeted 
Environments 


A System to Evaluate and Assess 
Holistic Aircrew Workload 


Stand-Off Biometric Assay 


Assessment and Diagnosis of 
Vestibular Indicators of Soldiers 
Operational Readiness 


Rapid Ethnographic Assessment 
and Data Integration Toolkit 


Framework for Sensing and 
Representing Negative Effects of 
Motion 


Project Theme 


Cyber teams / cognitive load, stress, 
fatigue, team dynamics, taskload 
allocation 


Medical teams / cognitive load 


T&E subjects / Mission analog 
environments 


Real-time assessment of Airmen 
human readiness level (HRL) 


Real-time assessment of novice test 
subject and prediction of state change 
after development of expertise 


Stand-off detection of HR 


Screening for vestibular damage or 
Mild Traumatic Brain Injury 


Aid capture of social network data 


Sense sopite syndrome 


e Company background 
e Company overview 
e Human sensing projects overview 


¢e Cognitive workload sensing 
e Why do we care? 
¢ Who cares? 


e Methods 
e Custom sensing devices 
e Data handling 
e Information display 


e Conclusions 
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The Need 


1. Many work environments fraught with highly 
variable demands on cognitive workload 


e Fluctuation between cognitive overload and boredom 
° Detrimental to performance | | 
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(@orelanian= Assessment and Prediction to Promote Individualized 
Capability Augmentation and Reduce Decrement (CAPT PICARD) 


CAPT PICARD shetlock 


Data Fusion and Interpretation Engine 


Raw Data Bayesian Networks, Causal Influence 
Streams 


Me oY Models, Weighted Averaging 
F Pm, S Processed 1 7 ) J 


iP er orehéad Sensor suite Data Streams Physical Health Sleep 
Cognitive Workload Fatigue 

Mood 

Stress 


Targeted Human Performance Metrics 


| 


Performance Prediction 
Engine 


Deep Learning 


- Sum Product Network 


- Neural Networks 
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The Need 


2. Fast-paced, sometimes dangerous professions 
require personnel to be trained to perform duties 
automatically 


e If skill still requires great effort, additional training is 
necessary 


e Focus training to enhance learning 
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Data Fusion Engine 


Raw Data Streams 
Time-Alignment, Data Filtering, 
Data Fusion, Data Processing 


After-Action User Interface 


After Action Review of Entire Simulation Sherlock’ 
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The Need 


3. Teams of individuals must focus on their own tasks, 
while maintaining awareness of teammates’ tasks 


e Optimize team performance on 


Adaptive Toolkit for the Assessment and Augmentation of 


Performance by Teams in Real Time (ADAPTER) 


Intelligent 
Experimenter Interface 


Task 
Demands 
and 


Data Fusion Toolset 
Sensor Operator State | Team State 
Architecture Data Fusion Data Fusion 
State metrics 


Model-Building and Executi6n Toolset 


pa Real-Time Assessment 


Model Execution Tool descriptions. | Uses fused and modeled 
Model selected or created by user and data to assess operator 
and populated with output from the and team states in real-time 
Data Fusion Toolset Optimal by comparing current states 


state to ideal states 
descriptions 


Real-Time Augmentation 


« Compares operator and 
team states to those 
optimal for performance, 
based on currently 
implemented model 
Determines which metrics 
need to be augmented to 
optimize performance 


Adaptation View 


« More tasks to 
Operator #1 

« Fewer tasks to 
Operator #2 

« Disable voice chat 

« Enable text chat 
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The Need 


4. As new tools and systems are developed, their 
effect on cognitive workload must be assessed 


e Interaction should be intuitive; if additional workload is 
unacceptable, design must be adapted throughout 
developmental phases 


e Testing and evaluation in laboratory experiments 
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Sherlock™ 


Open, extensible software and hardware platform for 
rapid prototyping of solutions to collect, analyze, 
visualize, and reason about human physiological, 
neurological, and behavioral state 


e COTS and custom sensors (patent pending) 

e Data processing and analysis 

¢ Classification of human states (e.g., cognitive load) 
e Statistical and probabilistic modeling techniques 

¢ Prediction of performance 
e Machine-learning algorithms, hierarchical modeling 


charles river analytics 


Custom Portable fNIRS Sensor 


| Mr. Filipe Silva, Biosignals PluX , , 


Sensor Validation 


¢ Simultaneous EEG/fNIRS data collection 


e Compare attention/workload measures of both 
sensors 


Dr. Elena Festa, Brown University 
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MEDIC Il - b 


Scenario Name: Bethany Test 


Start Time: 09:57:02 09/29/2016 © 


0 45 90 135 180 225 270 300 


Duration (minutes): 


Workload Rating Interval (Min) (mi... Workload Rating Interval (Max) (m... 


Interval Time Out Limit (minutes): Rating Time Out Limit (minutes): 


Add Subject 


No content in table 


Cancel 


Mr. Alex Negri 
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MEDIC II - bbracken 


Start Time: 11:36:18 End Time: 00:00:00 Time: 12:32:33 


Stop Scenario 


12:06 PM 12:21 PM 


Team Notes Field Notes 


Workload Ratings 


+ Workload Ras + Field ... + Subject ... + Proce... + Cas... 


Role 


Subject Notes 


Soldier 


Subject 1 Hot Seat 1 


Subject 2 First Responder 


Subject 3 Observer 


Mr. David Koelle 
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Data Fusion 


Heart Rate Var. 
| IR Imaging | 
| Motion Cap. 


EEG 
Analytic Beta Waves 
Plugin 


EEG 


EMG Heart Rate Variability 
Plugin avi 
MG 
Vv 
Myon Biosignal Emotiv FLIR PrioVR 
Hardware Hardware Hardware Hardware Hardware 
Plugin Plugin Plugin Plugin Plugin 


PrioVR 
Motion 
Cap. 


Imaging 
cvorv ES 


Mr. Scott Irvin 
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Estimating Cognitive Workload 


Movement 
Oxygenation 


pNIRS 
Heart Rate 
(33%) Average Blood yg 
Deoxygenation 


M1 Physical 
Activity ee 


(47%) M1 Cognitive 
Load Prediction 


Ms. Noa Palmon 
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Pilot Study: Estimating Cognitive Workload 


Two Variable CIM 
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Time (seconds) 


Three Variable CIM 
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Assessing Cognitive Overload and Boredom 


e Baseline Rest 
e Cognitive Task Battery 
e N-back (3 levels) 
e Digit Symbol Substitution 
e Psychomotor Vigilance 
e NASA Multi-Attribute Task Battery (3 levels) 
¢ Boredom Induction 
e Peg Turning Task 
e Repeat Cognitive Task Battery 
e Post Rest 


4 Dr. Elena Festa, Brown University 
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Assessing Cognitive Overload and Boredom 
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Dr. Elena Festa, Brown University 
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Assessing Cognitive Overload and Boredom 


¢ Boredom induction (Mackey et al., 2014) 
e Peg turning 


O2d00 
OOOO 


o Dr. Elena Festa, Brown University 
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Data Fusion Engine 


Raw Data Streams 
Time-Alignment, Data Filtering, 
Data Fusion, Data Processing 


After-Action User Interface 


After Action Review of Entire Simulation Sherlock’ 
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Assessing Cognitive Overload During Physical 


AN URAL AY, 


¢ Baseline e Hot Potato 

¢ Word list learning e Logic problems 
¢ Balance ball e Moving Boxes 

e 20 Questions e Word list recall 
e Puzzle e Team jump rope 


¢ 19 teams completed (excluding pilot data and 
drop-outs) for a total of 57 participants 


® Dr. Polemnia Amazeen, Arizona State University 
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Data Processing During Motion 


Data Viewer 


, | | Charts Shown , | | Update Selections Folder: X:\Contracts\Contract Documents\FY 14 Contracts\C14046 MEDIC II\Data\ASU\All_Data\Processed\Team21_5_31_16 


Participant 1 Helmet Device 


— Participant 2 Helmet Device 
—— Participant 3 Helmet Device 


Dr. Seth Elkin-Frankston 
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Data Processing During Motion 
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Data Processing During Motion 
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Data Processing During Motion 


/ GH Data Viewer x 
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Estimating Cognitive Workload During Physical 


Activity 


e Will insert medic modeled data here if we get it done 
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Estimating Cognitive Workload During Physical 


PANU AVA LAY, 


e Teams of medical students and faculty 
members 


¢ High-fidelity medical simulations 
¢e Will update with appropriate data collection 
Statistics (include the data we collected if we 


can use it, if not estimate when collection will 
Start) 


t ) Dr. Matt Weinger, Vanderbilt Medical School 
fe 


Dr. Arna Banerjee, Vanderbilt Medical School 
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Adaptive Toolkit for the Assessment and Augmentation of 


Performance by Teams in Real Time (ADAPTER) 


Intelligent 
Experimenter Interface 


Task 
Demands 
and 


Data Fusion Toolset 
Sensor Operator State | Team State 
Architecture Data Fusion Data Fusion 
State metrics 


Model-Building and Executi6n Toolset 


pa Real-Time Assessment 


Model Execution Tool descriptions. | Uses fused and modeled 
Model selected or created by user and data to assess operator 
and populated with output from the and team states in real-time 
Data Fusion Toolset Optimal by comparing current states 


state to ideal states 
descriptions 


Real-Time Augmentation 


« Compares operator and 
team states to those 
optimal for performance, 
based on currently 
implemented model 
Determines which metrics 
need to be augmented to 
optimize performance 


Adaptation View 


« More tasks to 
Operator #1 

« Fewer tasks to 
Operator #2 

« Disable voice chat 

« Enable text chat 
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Team-Specific Cognitive Workload Metrics 


¢ Teams of three unmanned aerial vehicle (UAV) task: 
¢ Air Vehicle Operator (avo) flies the UAV 
¢ Payload Operator (plo) takes photos of targets 
¢ Data exploitation, mission planning, and 
communications operator (dempc) uses a map to plan 
the UAV route and coordinates with the other 
teammates 


. Dr. Nancy Cooke, Arizona State University 
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Team-Specific Cognitive Workload Metrics 


¢ Measures of importance are task dependent 
¢ Inter-subject correlation for blood oxygenation 
significant predictor of team performance (R2 = .08, 
p < .05) 
¢ But ONLY for the team leader 
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Predicting Cognitive Workload and Performance 


e Design and train classifiers to predict shape of 
cognitive workload curve (and thus predict when 
performance will decay) several time steps (seconds 
or minutes) in advance 
1. Design classifiers to distinguish between groups 
(bored vs. baseline) and between tasks (e.g., easy, 
medium, hard) 

2. Curve estimation based on initial portion of curve to 
predict upcoming decrements 
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Classifier Design 
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Curve Estimation 
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¢ Cognitive workload sensing 
e Why do we care? 
e Who cares? 


e Methods 
e Custom sensing devices 
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e Information display 


« Conclusions 
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Generalization of Results 


e We believe this approach is widely applicable 
across domains 


¢e Currently researching commercial applications: 
e Marketing 
e Finance 
e Education 
e Gaming 
e Psychiatric monitoring (e.g., children with ADHD) 
e Always looking for collaboration opportunities! 
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